Control apparatus, method for controlling control apparatus, server, controlled apparatus, control system, and storage medium storing control program

ABSTRACT

A smartphone includes: an image display section for displaying images in such a manner as to be selectable by a user, each of the images being associated with (i) specifying information for specifying a cleaning robot and (ii) instruction information for instructing the cleaning robot to execute a predetermined process; and a process instruction section for transmitting, in a case where the user selects at least one of the images thus displayed, image information (2) indicative of said at least one of the images to a server, so as to cause the cleaning robot specified by specifying information associated with said at least one of the images to execute a predetermined process that is instructed by instruction information associated with said at least one of the images.

This Nonprovisional application claims priority under 35 U.S.C. §119 onPatent Application No. 2012-219848 filed in Japan on Oct. 1, 2012, theentire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to apparatuses such as a control apparatusfor controlling a controlled apparatus via a server.

BACKGROUND ART

For years, researches have been conducted on techniques related to amethod of operating an external apparatus via a predetermined interface.Particularly, since the operation of the external apparatus has beenmade complicated along with advancement of processes executable by theexternal apparatus, many attempts have been carried out in recent yearsin order to improve the interface.

For example, such an interface has been developed via which a processcan be specified by a simple illustration (image). Patent Literature 1discloses an image data receiving device for receiving, from theoutside, an image data set storing therein information (processspecifying information) for specifying a process which should be carriedout by the image data receiving device itself. With this, the image datareceiving device can be operated freely even by an operation device onlyhaving an image data transmission function.

CITATION LIST Patent Literatures

[Patent Literature 1]

Japanese Patent Application Publication, Tokukai, No. 2009-118448 A(Publication Date: May 28, 2009)

SUMMARY OF INVENTION Technical Problem

According to the technique disclosed by Patent Literature 1, the imagedata set includes only the “process specifying information”. Therefore,a user must specify the external device which is to be operated. Namely,the user is forced to carry out an explicit operation of specifying anapparatus which is to be operated and transmitting an instruction to theapparatus thus specified (for example, an operation of pointing aninfrared ray-transmitting part of a remote controller at a television).

However, the interface that forces the user to carry out such theexplicit operation is not user-friendly. What is truly required by theuser is a “function” that is provided by the external apparatus. In somecases, the apparatus that plays a main role in order to provide the“function” may be arbitrary one.

The present invention was made in view of the above problem, and has anobject to provide a control apparatus and the like each enabling a userto operate an external apparatus which is to be operated even if theuser does not carry out an explicit operation of specifying the externalapparatus, for the purpose of improving user's convenience.

Solution to Problem

In order to solve the foregoing problem, a control apparatus accordingto one embodiment of the present invention is a control apparatus forcontrolling a controlled apparatus via a server, including: firstdisplay means for displaying images in such a manner as to be selectableby a user, each of the images being associated with (i) a piece ofspecifying information for specifying a controlled apparatus and (ii) apiece of instruction information for instructing the controlledapparatus to execute a predetermined process; and instruction means fortransmitting, in a case where the user selects at least one of theimages displayed by the first display means, a piece of imageinformation indicative of said at least one of the images to the server,so as to cause a controlled apparatus that is specified by a piece ofspecifying information associated with said at least one of the imagesto execute a predetermined process that is instructed by a piece ofinstruction information associated with said at least one of the images.

In order to solve the foregoing problem, a method for controlling acontrol apparatus according to one embodiment of the present inventionis a method for controlling a control apparatus that controls acontrolled apparatus via a server, including: a first displaying step ofdisplaying images in such a manner as to be selectable by a user, eachof the images being associated with (i) a piece of specifyinginformation for specifying a controlled apparatus and (ii) a piece ofinstruction information for instructing the controlled apparatus toexecute a predetermined process; and an instruction step oftransmitting, in a case where the user selects at least one of theimages displayed in the first displaying step, a piece of imageinformation indicative of said at least one of the images to the server,so as to cause a controlled apparatus that is specified by a piece ofspecifying information associated with said at least one of the imagesto execute a predetermined process that is instructed by a piece ofinstruction information associated with said at least one of the images.

Advantageous Effects of Invention

According to the one embodiment of the present invention, the controlapparatus and the method for controlling the control apparatus eachenable a user to control a controlled apparatus which is to becontrolled even if the user does not carry out an explicit operation ofspecifying the controlled apparatus. With this, the control apparatus orthe like provides improved user's convenience.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a main part of a configuration ofa smartphone according to one embodiment of the present invention.

FIG. 2 is a block diagram illustrating a main part of a configuration ofa server according to one embodiment of the present invention.

(a) of FIG. 3 is a block diagram illustrating a main part of aconfiguration of a cleaning robot according to one embodiment of thepresent invention.

(b) of FIG. 3 is a block diagram illustrating a main part of aconfiguration of a television image receiver according to one embodimentof the present invention.

FIG. 4 shows an outline of a family message board system according toone embodiment of the present invention.

(a) of FIG. 5 is a view schematically illustrating an example of anappearance of the smartphone.

(b) of FIG. 5 is a view schematically illustrating an example where thesmartphone is operated by a user.

(a) and (b) of FIG. 6 are views each schematically illustrating howmessages and images are exchanged between a user and the cleaning roboton the family message board.

(a) and (b) of FIG. 7 are views each schematically illustrating anexample of a screen showing a list of images selectable by a user.

FIG. 8 is a view schematically illustrating the family bulletin board onwhich the cleaning robot contributes a result of execution of a seriesof processes specified by a plurality of images contributed by a user atonce.

FIG. 9 is a flow chart of an example of a process executed in the familymessage board system.

FIG. 10 is a flow chart of another example of a process executed in thefamily message board system.

DESCRIPTION OF EMBODIMENTS

With reference to FIGS. 1 through 10, embodiments of the presentinvention will be described in detail below.

[Outline of Family Message Board System 400]

With reference to FIG. 4, the following describes an outline of a familymessage board system 400 according to one embodiment of the presentinvention. FIG. 4 shows the outline of the family message board system400.

The family message board system (control system) 400 is a system (aso-called social network service) providing an electronic message boardthat is shareable by users (in the example shown in FIG. 4, a father, amother, and a child) having been registered in advance. The familymessage board system 400 includes smartphones 100 a, 100 b, and 100 c,an apparatus controlling server 200 a, a family message board server 200b, a home server 200 c, a cleaning robot 300 a, and apparatuses locatedin a home (hereinafter, such the apparatuses are referred to as“domestic apparatuses”).

In the family message board system 400, the cleaning robot 300 a or anyof the domestic apparatuses writes a message on a family message boardaccording to the state of the domestic apparatus and/or the content of amessage written by any of the users. Further, any of the users canaccess the family message board so as to write or browse a messagethereon/therefrom by use of one of the smartphones 100 a, 100 b, and 100c that is in the user's possession. Furthermore, any of the users cancontrol operation of any of the domestic apparatuses by contributing animage to the family message board (this will be described in detaillater with reference to FIG. 6).

The home server 200 c is communicably and wirelessly connected to thecleaning robot 300 a. Further, the home server 200 c is communicably andwirelessly connected (or wire-connected) to the domestic apparatuses.FIG. 4 shows the example where the domestic apparatuses are a television300 b, an air conditioning apparatus 300 c, an illumination apparatus300 d, and a video recording apparatus 300 e. However, the types and thenumbers of domestic apparatuses are not limited to those in thisexample.

As a personified character of the cleaning robot 300 a, the apparatuscontrolling server 200 a writes a message and/or the like on the familymessage board according to, e.g., the state of any of the domesticapparatuses, information obtained by any of the domestic apparatuses,and/or the content of a message written by a user. This allows the userto simulatively experience communication with the cleaning robot 300 aon the family message board.

The apparatus controlling server 200 a accepts an image contributed byany of the smartphones 100 a, 100 b, and 100 c, and transmitsinstruction information associated with the image to the home server 200c, so as to control operation of a corresponding one of the cleaningrobot 300 a and the domestic apparatuses.

Note that the present embodiment deals with an example where theapparatus controlling server 200 a has (i) a function of writing amessage and/or the like on the family message board and (ii) a functionof controlling operation of the cleaning robot 300 a and the domesticapparatuses. However, these functions may be provided in separateservers.

The family message board server 200 b carries out all processesregarding the family message board such as (i) providing a display ofthe family message board with each of the smartphones 100 a, 100 b, and100 c and (ii) management of a message and/or the like written on thefamily message board.

The home server 200 c comprehensively controls exchange of informationbetween (i) the cleaning robot 300 a or any of the domestic apparatusesand (ii) the apparatus controlling server 200 a. Specifically, the homeserver 200 c controls operation of the cleaning robot 300 a or any ofthe domestic apparatuses according to instruction information suppliedfrom the apparatus controlling server 200 a. Further, the home server200 c transmits, to the apparatus controlling server 200 a, informationobtained from the cleaning robot 300 a and information obtained from thedomestic apparatuses.

The air conditioning apparatus (controlled apparatus) 300 c is anapparatus for carrying out air conditioning such as heating and cooling(hereinafter, such the apparatus is also referred to as an “airconditioner 300 c”). The air conditioning apparatus 300 c includes atemperature sensor, and transmits a detected room temperature to thehome server 200 c.

The illumination apparatus (controlled apparatus) 300 d is anillumination apparatus including a light source such as an LED. Undercontrol of the home server 200 c, the illumination apparatus 300 d canbe turned on and off.

The television (controlled apparatus) 300 b is a television imagereceiver (hereinafter, such the apparatus is also referred to as a“television image receiver 300 b”). The video recording apparatus(controlled apparatus) 300 e is an apparatus for recording abroadcasting program received by the television 300 b. The television300 b and the video recording apparatus 300 e can also be controlled bythe home server 200 c.

In the example shown in FIG. 4, all the domestic apparatuses areincluded in the network, and are communicably connected to the homeserver 200 c. However, since the cleaning robot 300 a is capable ofrelaying instruction information transmitted from the home server 200 cto any of the domestic apparatuses, the domestic apparatuses may not bedirectly connected to the home server 200 c.

For example, even in a case where the television image receiver 300 b isnot included in the network, causing the cleaning robot 300 a totransmit the instruction information to the television image receiver300 b through IrDA or infrared rays allows the television image receiver300 b to execute a process included in the instruction information(e.g., a process of turning on the power or a process of switching thechannel). An example of controlling the television image receiver 300 bwhich is not included in the network will be described in detail later.

In the example shown in FIG. 4, the home server 200 c transmits sensingdata obtained by a sensor provided in the cleaning robot 300 a or any ofthe domestic apparatuses, an operation log of any of the domesticapparatuses, and a photograph taken by any of the domestic apparatuses.However, information to be transmitted by the home server 200 c is notlimited to these. Furthermore, in a case where the comprehensive controlby the home server 200 c is not necessary (e.g., in a case where nodomestic apparatus related to the family message board exists), thefamily message board system 400 may be configured such that no homeserver 200 c is provided and the cleaning robot 300 a exchangesinformation with the apparatus controlling server 200 a.

[Outline of Smartphone 100]

With reference to FIG. 5, the following schematically describes (i) anexample of an appearance of a smartphone 100 and (ii) an example wherethe smartphone 100 is operated. (a) of FIG. 5 is a view schematicallyillustrating the example of the appearance of the smartphone 100. (b) ofFIG. 5 is a view schematically illustrating the example where thesmartphone is operated by a user. Note that each of the smartphones 100a, 100 b, and 100 c shown in FIG. 4 has the same configuration as thatof the smartphone 100 shown in FIG. 5.

The smartphone (control apparatus) 100 is an information terminal forcontrolling the cleaning robot 300 a, the television image receiver 300b, and the like via the server 200.

As shown in (a) of FIG. 5, the smartphone 100 includes an input screen41, a display screen 72, and a power switch 43.

As shown in (b) of FIG. 5, by touching the input screen 41 so as tospecify a position in the input screen 41 with respect to the smartphone100 (i.e., to give the smartphone 100 coordinates on the input screen41), the user can give the smartphone 100 information of imageselection, for example.

As described above, the smartphone 100 is the potable informationterminal. Note here that the same functions of the smartphone 100 can beattained by, e.g., a mobile phone, a personal computer, or a tabletterminal. That is, the terminal may not be the smartphone, as long asthe terminal is an apparatus capable of inputting and outputtingnecessary information.

[Outline of Server 200]

The server 200 is communicably connected to the smartphone 100, thecleaning robot 300 a, and the domestic apparatuses. Namely, thefollowing description will be made based on the assumption that the“server 200” comprehensively has the substantial functions of theapparatus controlling server 200 a, the family message board server 200b, and the home server 200 c shown in FIG. 4.

[Outline of Cleaning Robot 300 a]

The cleaning robot (controlled apparatus) 300 a is one of controlledapparatuses controlled by the smartphone 100 via the server 200. Thecleaning robot 300 a has a function of carrying out cleaning while thecleaning robot 300 a is travelling.

The cleaning robot 300 a further has, e.g., a function of storing anoperation log, a function of detecting and outputting a remaining chargelevel, a function of taking an image, a function of recognizing a voice,and a function of outputting a voice. Further, the cleaning robot 300 ahas a function of transmitting a control signal to any of the domesticapparatuses so as to operate the domestic apparatus.

[Control of Apparatus by Contribution of Image 5]

With reference to FIG. 6, the following schematically describes a flowof a series of processes executed in the family message board system 400in order to control the cleaning robot 300 a or any of the domesticapparatuses in response to an image 5 contributed on the family messageboard by the user. (a) and (b) of FIG. 6 are views each schematicallyillustrating how messages and images 5 are exchanged between a user(mother) and the cleaning robot 300 a on the family message board.

As shown in (a) of FIG. 6, the cleaning robot 300 a measures a roomtemperature with a temperature sensor, and contributes, on the familymessage board, a message 8 a telling that “it is hot” in human sense.Upon reading this message, the user contributes, on the family messageboard, an image 5 instructing the air conditioner 300 c to turn on anair-cooling function as a message 8 b.

Namely, in response to user's touch on an image selecting button 44, thesmartphone 100 displays a plurality of images selectable by the user,and transmits to the server 200 image information 2 indicative of theimage 5 selected by the user among the images.

Note that the image 5 is associated with (i) specifying information 1 afor specifying a control target (controlled apparatus) and (ii)instruction information 1 b for instructing the control target toexecute a predetermined process. In the example shown in (a) of FIG. 6,the specifying information 1 a, which is associated with the image 5,indicates the “air conditioner 300 c”, and the instruction information 1b indicates the instruction to “turn on an air-cooling function”.

The image information 2 is information by which the image 5 can beuniquely identified. The image information 2 may take any form, as longas the image 5 can be uniquely identified by the image information 2.For example, the image information 2 may be constituted by a numeral(s)or a character(s). In a case where the smartphone 100 and the server 200have the same directory structure for storing the image 5, the imageinformation 2 may be a name of a path for storing the image.Alternatively, the image information 2 may be the image 5 itself (inthis case, the smartphone 100 transmits the image 5 to the server 200).

Here, upon reception of the image information 2, the server 200transmits the message 8 b to the plurality of smartphones 100 ascontribution information 4 c, in order to allow information that theinstruction has been given by the smartphone 100 with use of the imageinformation 2 to be shared by users of other smartphones. Note that themessage 8 b (contribution information 4 c) may include, in addition tothe image 5, information that is related to the message 8 b and is addedby the server 200 (e.g., information by which the user contributed themessage 8 b is identified and/or the date and time of the contribution).

Further, upon reception of the image information 2, the server 200transmits, to the apparatus specified by the specifying information 1 aassociated with the image 5 indicated by the image information 2, theinstruction information 1 b for instructing the apparatus to execute thepredetermined process. In the example shown in (a) of FIG. 6, the server200 transmits, to the “air conditioner 300 c” specified by thespecifying information 1 a, the instruction information 1 b forinstructing the air conditioner 300 c to “turn on the cooling function”.

Upon reception of the instruction information 1 b, the air conditioner300 c executes the process of “turning on the cooling function”. Afterthat, the cleaning robot 300 a contributes, on the family message board,a message 8 c telling that he is happy about the room being cooled.

As shown in (b) of FIG. 6, the cleaning robot 300 a records a voice of apet taken with a microphone. Further, on the family message board, thecleaning robot 300 a contributes (i) a message 8 d informing that thepet is calling and (ii) an audio file. Upon reading the message andlistening to the recorded voice, a user contributes, on the familymessage board, an image 5 instructing the cleaning robot 300 a to take aphotograph of the surroundings as a message 8 e.

Namely, as well as in the above-described operation, in response touser's touch on the image selecting button 44, the smartphone 100displays a plurality of images selectable by the user, and transmits tothe server 200 image information 2 indicative of the image 5 selected bythe user among the images.

Upon reception of instruction information 1 b for giving an instructionto “take a photograph of the surroundings” via the server 200, thecleaning robot 300 a executes the process of taking a photograph, andcontributes, on the family message board, (i) a taken image (result ofexecution of the process) and (ii) a message 8 f informing that thecleaning robot 300 a has taken the photograph.

Here, the smartphone 100 may display, in the vicinity of the message 8e, the taken image obtained from the cleaning robot 300 a via the server200. With this, the smartphone 100 can allow the user to recognize thatthe taken image and the message 8 f are a response of the cleaning robot300 a to the message 8 e (i.e., association between the message 8 e andthe result of execution of the process).

To summarize the above-described outline, the smartphone 100 executes aprocess in the following steps of: (1) displaying images 5 in such amanner as to be selectable by a user, each of the images 5 beingassociated with (i) a piece of specifying information 1 a for specifyinga corresponding one of the cleaning robot 300 a and the domesticapparatuses and (ii) a piece of instruction information 1 b forinstructing the corresponding one of the cleaning robot and the domesticapparatuses to execute a predetermined process; and (2), in a case wherethe user selects at least one of the images 5 thus displayed,transmitting a piece of image information 2 indicative of the at leastone of the images 5 to the server 200, so as to cause any of thecleaning robot and the domestic apparatuses specified by the piece ofspecifying information 1 a to execute a predetermined process specifiedby the piece of instruction information 1 b.

According to the above-described conventional technique, an imageincludes only the instruction information, and therefore a user mustspecify an external apparatus which is to be operated. Namely, the useris forced to carry out an explicit operation of specifying the apparatuswhich is to be operated and transmitting an instruction to the apparatusthus specified.

However, what is truly required by the user is a “function” that isprovided by the external apparatus. In some cases, the apparatus thatplays a main role in order to provide the “function” may be arbitraryone. Thus, the interface always forcing the user to carry out such theexplicit operation is not user-friendly.

On the other hand, the smartphone 100 transmits, to the server 200, theimage information 2 indicative of the image 5 associated with thespecifying information 1 a and the instruction information 1 b. Namely,since the smartphone 100 always transmits the image information 2 to the“server 200”, the smartphone 100 does not require the user to carry outthe explicit operation of “specifying the external apparatus”.

In other words, the smartphone 100 allows the user to control thecleaning robot or any of the domestic apparatuses even if the user doesnot carry out the explicit operation of specifying the cleaning robot300 a or any of the domestic apparatuses, which is a control target.With this, the smartphone 100 provides improved user's convenience.

[Configuration of Smartphone 100]

With reference to FIG. 1, the following describes a configuration of thesmartphone 100. FIG. 1 is a block diagram illustrating a main part ofthe configuration of the smartphone 100. Note here that parts being notdirectly related to the present embodiment (e.g., a part enabling atelephone call through a telephone line) are omitted in the followingdescription of the configuration and the block diagram in FIG. 1.However, according to the actual condition in which the smartphone 100is worked, the smartphone 100 may include the omitted parts.

In order to simplify the description, the following description of theblock diagram will be made based on the assumption that the familymessage board system 400 (see FIG. 4) only includes the smartphone(control apparatus) 100, the server 200, the cleaning robot (controlledapparatus) 300 a, and the television image receiver (controlledapparatus) 300 b. Further, the description will be made also based onthe assumption that the television image receiver 300 b is not includedin the network and cannot receive the instruction information 1 b fromthe server 200.

In the following description, functions of an input section 40 (an inputscreen 41 and an input control section 42), a control section 10 (aprocess instruction section 11, an execution possibility judging section12, an execution result obtaining section 13, and a contributioninformation obtaining section 17), a communication section 20 a (atransmitting section 21 a and a receiving section 22 a), and a displaysection 70 (a display control section 71 and a display screen 72) willbe explained in this order.

The input section 40 accepts an operation carried out by a user (useroperation 7 a). The input section 40 includes the input screen 41 andthe input control section 42.

The input screen 41 outputs, to the input control section 42, an inputsignal 7 b indicative of a position on the input screen 41 whichposition is specified by the user operation 7 a. In the presentembodiment, the input screen 41 is preferably a touch panel capable ofdetecting multi-touch operation. However, the type of hardware of theinput screen 41 is not limited to the touch panel, as long as the inputscreen 41 is an input device capable of detecting the position of thetouch made by the user operation 7 a.

The input control section 42 generates input information 7 c based onthe input signal 7 b supplied from the input screen 41, and outputs theinput information 7 c to an image display section 71 a and the processinstruction section 11. Here, the input information 7 c includes atleast (i) information that the image selecting button 44 has beentouched by the user and (ii) information that an image 5 has beenselected by the user.

The control section 10 comprehensively controls various kinds offunctions of the smartphone 100. The control section 10 includes theprocess instruction section 11, the execution possibility judgingsection 12, the execution result obtaining section 13, and thecontribution information obtaining section 17.

In a case where at least one of images 5 displayed by the image displaysection 71 a is selected by the user, the process instruction section(instruction means) 11 transmits image information 2 indicative of theselected image 5 to the server 200, so as to cause the cleaning robot300 a or the domestic apparatus specified by specifying information 1 aassociated with the selected image 5 to execute a predetermined processspecified by the instruction information 1 b.

Specifically, upon receiving, from the input control section 42, theinput information 7 c indicating that the image 5 is selected by theuser, the process instruction section 11 outputs the image information 2indicative of the selected image 5 to the transmitting section 21 a.

Note that, in a case where a plurality of ones of the images 5 areselected by the user, the process instruction section 11 transmits, tothe server 200, a plurality of pieces of image information 2 indicativeof the respective plurality of ones of the images 5, so as to cause thecleaning robot 300 a and/or the domestic apparatus specified by piecesof specifying information 1 a associated with the respective pluralityof ones of the images 5 to sequentially execute predetermined processesspecified by pieces of instruction information 1 b associated with therespective plurality of ones of the images 5. This process will bedescribed in detail later.

Upon reception of apparatus information 3 b from the receiving section22 a, the execution possibility judging section (first judging means) 12judges whether or not the cleaning robot 300 a or the domestic apparatuscan execute the predetermined process, according to the apparatusinformation 3 b. Then, the execution possibility judging section 12outputs a judgment result 6 a thereof to the process instruction section11 and the image display section 71 a.

The apparatus information 3 b includes information related to a functionof an apparatus and a process that the apparatus is executing. Forexample, the apparatus information 3 b includes (i) information that thevideo recording apparatus 300 e has such a function of recording aprogram and another program in the same time slot on a differentchannel, (ii) information that some functions of the air conditioner 300c are disabled because a filter of the air conditioner 300 c is soiled,or (iii) information that the cleaning robot 300 a is taking aphotograph.

The execution possibility judging section 12 judges whether or not theapparatus can execute the predetermined process specified by theinstruction information 1 b associated with the image 5 under the stateof the apparatus indicated by the apparatus information 3 b. Forexample, in a case where the apparatus information 3 b indicates that“the cleaning robot 300 a is taking a photograph”, the executionpossibility judging section 12 determines that the cleaning robot 300 acannot execute a process of “cleaning”.

The execution result obtaining section (obtaining means) 13 obtains aresult (execution result 3 a) of execution of the predetermined processcarried out by the cleaning robot 300 a or the domestic apparatus fromthe server 200 via the receiving section 22 a, and outputs the executionresult 3 a to a list display section 71 b.

For example, in a case where the user causes the cleaning robot 300 a toexecute a process of “taking a photograph of the surroundings”, theexecution result 3 a is an image that is taken by the cleaning robot 300a and transmitted via the server 200.

In a case where contribution information 4 c is supplied from thereceiving section 22 a to the contribution information obtaining section17, the contribution information obtaining section 17 outputs thecontribution information 4 c to the list display section 71 b. Note herethat the contribution information 4 c is information including themessage 8 b (see (a) of FIG. 6) so as to allow information that theinstruction is given by one smartphone 100 with use of the imageinformation 2 to be shared by users of other smartphones, the message 8b being transmitted from the server 200 which has received the imageinformation 2.

The communication section 20 a communicates with an entity on theoutside via a communication network that complies with a predeterminedcommunication protocol. A communication line, a communication mode, or acommunication medium is not particularly limited, as long as it has asubstantial function for enabling communication with an externalapparatus. The communication section 20 a can be made of, e.g., a devicesuch as an Ethernet (Registered Trademark) adapter. The communicationsection 20 a can use, e.g., a communication mode and/or a communicationmedium such as IEEE 802.11 wireless or Bluetooth (Registered Trademark).The communication section 20 a includes the transmitting section 21 aand the receiving section 22 a.

The transmitting section 21 a transmits, to the server 200, the imageinformation 2 supplied from the process instruction section 11.

The receiving section 22 a receives the execution result 3 a, theapparatus information 3 b, and the contribution information 4 c each ofwhich is transmitted from the server 200, and outputs the executionresult 3 a, the apparatus information 3 b, and the contributioninformation 4 c to the execution result obtaining section 13, theexecution possibility judging section 12, and the contributioninformation obtaining section 17, respectively.

The display section 70 shows, to the user, the family message board andcontents included in the family message board (e.g., a message and/or animage). The display section 70 includes the display control section 71and the display screen 72.

The display control section 71 outputs, to the display screen 72,display data 7 d so that the family message board and the contentsincluded in the family message board are displayed on the display screen72. The display data 7 d may be, for example, an image according to abitmap format, an image according to any other format, or data accordingto a format suitable for displaying. The display control section 71includes the image display section 71 a and the list display section 71b.

The image display section (first display means) 71 a displays the images5 in such a manner as to be selectable by the user, each of the images 5being associated with the specifying information 1 a and the instructioninformation 1 b. Specifically, upon receiving, from the input controlsection 42, the input information 7 c indicating that the imageselecting button 44 is touched by the user, the image display section 71a displays the plurality of images 5 selectable by the user.

In a case where a judgment result 6 a indicating that a predeterminedprocess specified by instruction information 1 b associated with one ofthe images 5 cannot be executed by the specified apparatus is suppliedfrom the execution possibility judging section 12 to the image displaysection 71 a, the image display section 71 a does not display the one ofthe images 5 associated with that predetermined process, or displays theone of the images 5 in a reduced brightness.

With reference to FIG. 7, the following describes an example of aninterface for selection displayed on the display screen 72 in the casewhere the images 5 are displayed in such a manner as to be selectable.(a) and (b) of FIG. 7 are views each schematically illustrating anexample of a screen showing a list of images selectable by a user.

As shown in (a) of FIG. 7, the smartphone 100 (image display section 71a) displays the images 5 in such a manner as to be viewed at a glance(e.g., in a list), each of the images 5 expressing a process that theuser can instructs the cleaning robot 300 a to execute. As describedabove, according to the state of the cleaning robot 300 a (i.e., whetheror not the cleaning robot 300 a can execute the processes expressed bythe images 5), the display of the list is changed.

For example, in a case where the apparatus information 3 b indicatesthat “the cleaning robot 300 a is taking a photograph” and therefore theexecution possibility judging section 12 determines that the cleaningrobot 300 a cannot execute a process of “cleaning”, an option to “causethe cleaning robot to carry out cleaning” is displayed in anunselectable manner (e.g., this option is not displayed or is displayedin a reduced brightness).

In a case where the user attempts to select the image 5 displayed in theunselectable manner, the cleaning robot 300 a may contributes, e.g., amessage “I will do it after five minutes. OK?” on the family messageboard, so as to prompt the user to wait for a while for execution.

As shown in (b) of FIG. 7, the smartphone 100 (image display section 71a) displays the images 5 in such a manner as to be viewed in a list,each of the images 5 expressing a process that the user can instruct adomestic apparatus (here, the television image receiver 300 b) toexecute.

Note here that the images displayed in the list are not changed independence on whether the television image receiver 300 b is in a state(1) in which the television image receiver 300 b is included in thenetwork and can receive the instruction information 1 b from the server200 or a state (2) in which the television image receiver 300 b is notincluded in the network and cannot receive the instruction information 1b from the server 200. The reason for this is as follows: Even in thestate (2), causing the cleaning robot 300 a to relay the instructioninformation 1 b to the television image receiver 300 b allows the userto control the television image receiver 300 b in a similar manner tothe state (1).

Namely, in a case where the user attempts to control the televisionimage receiver 300 b to turn on the television image receiver 300 b inthe state (2), the smartphone 100 can transmit, to the server 200, imageinformation 2 of the image 5 associated with (i) specifying information1 a specifying the “cleaning robot 300 a” and (ii) instructioninformation 1 b giving an instruction to “turn on the television imagereceiver 300 b”. Upon reception of the instruction information 1 b viathe server 200, the cleaning robot 300 a moves to a position at whichthe cleaning robot 300 a can transmit the instruction information 1 b tothe television image receiver 300 b through IrDA or infrared rays, andthen executes the process of “turning on the television image receiver300 b”.

In this manner, the cleaning robot 300 a relays the instructioninformation 1 b from the server 200 to the domestic apparatus. Thisallows the user to control that domestic apparatus via the smartphone100 without considering whether the domestic apparatus is the state (1)or the state (2). Note that, according to the state, a process ofchanging the specifying information 1 a and the instruction information1 b may be executed by an information extraction section 16 in theserver 200 (this will be described in detail later).

The list display section (second display means) 71 b displays, asinformation having been shared by the users, pieces of information intime series in such a manner as to be viewed at a glance, the pieces ofinformation including at least one piece of image information 2transmitted by the process instruction section 11 (see (a) and (b) ofFIG. 6). Note that, in a case where the execution result obtainingsection 13 obtains an execution result 3 a, the list display section 71b may further display the execution result 3 a in the vicinity of amessage and an image 5 that are displayed in such a manner as to beviewed in a list (see (b) of FIG. 6).

The display screen 72 displays the display data 7 d supplied from thedisplay control section 71. In the present embodiment, mainly assumed asthe display screen 72 is a liquid crystal display (LCD). However, thetype of hardware of the display screen 72 is not particularly limited,as long as the display screen 72 is an apparatus having a displayfunction (for example, the display screen 72 is an apparatus including adriver circuit for driving a display element based on the display data 7d supplied from the display control section 71).

In order to clearly indicate the functions of the respectiveconfigurations, the input screen 41 and the display screen 72 areseparately shown in FIG. 1. However, in a case where, for example, theinput screen 41 is a touch panel and the display screen 72 is a liquidcrystal display, the input screen 41 and the display screen 72 may beintegrated with each other as shown in (a) of FIG. 5. With thisconfiguration, (i) a position at which the user's finger comes intocontact with the input screen 41 coincides with (ii) a position atwhich, e.g., a figure is displayed in the display screen 72 in responseto the contact. As a result, the user can attain a natural feelingduring input operation.

[Configuration of Server 200]

With reference to FIG. 2, the following describes a configuration of theserver 200. FIG. 2 is a block diagram illustrating a main part of theconfiguration of server 200. In the following description, functions ofa communication section 20 b (a transmitting section 21 b and areceiving section 22 b), a communication possibility judging section 14,and the information extraction section 16 will be explained in thisorder.

The communication section 20 b has the same function as that of thecommunication section 20 a, and therefore detailed description of thecommunication section 20 b is omitted here. The communication section 20b includes the transmitting section 21 b and the receiving section 22 b.

The transmitting section (first transmitting means) 21 b transmits, tothe cleaning robot 300 a or the domestic apparatus specified byspecifying information 1 a associated with an image 5 indicated by imageinformation 2 received by the receiving section 22 b, instructioninformation 1 b giving an instruction to execute a predeterminedprocess.

Further, in order to check if the domestic apparatus is included in thenetwork and is in a state that the domestic apparatus can receive theinstruction information 1 b from the server 200 (i.e., whether thedomestic apparatus is in the state (1) or the state (2)), thetransmitting section 21 b transmits a response request 4 b to thedomestic apparatus. Further, in a case where the transmitting section 21b receives an execution result 3 a or apparatus information 3 b from thereceiving section 22 b, the transmitting section 21 b transmits theexecution result 3 a or the apparatus information 3 b to the smartphone100.

Furthermore, in order that a plurality of smartphones 100 can display amessage, the transmitting section 21 b generates a message including theimage 5 received by the receiving section 22 b (e.g., the message 8 b in(a) of FIG. 6), and transmits contribution information 4 c to all of theplurality of smartphones 100.

The receiving section (first receiving means) 22 b receives, from thesmartphone 100, the image information 2 indicative of the image 5selected by the user, and outputs the image information 2 to theinformation extraction section 16. In a case where the receiving section22 b receives an execution result 3 a or apparatus information 3 b fromthe cleaning robot 300 a, the receiving section 22 b outputs theexecution result 3 a or the apparatus information 3 b to thetransmitting section 21 b. In a case where the receiving section 22 breceives response information 4 a, the receiving section 22 b outputsthe response information 4 a to the communication possibility judgingsection 14.

Note here that the response information 4 a is information obtained inresponse to the response request 4 b that is transmitted from thetransmitting section 21 b to the cleaning robot 300 a or the domesticapparatus. Namely, the response information 4 a indicates that thecleaning robot 300 a or the domestic apparatus is included in thenetwork, and the cleaning robot 300 a or the domestic apparatus is in astate that the cleaning robot 300 a or the domestic apparatus canreceive the instruction information 1 b from the server 200 (theabove-described state (i)).

Specifically, in the case where the cleaning robot 300 a or the domesticapparatus is in the state (1), the receiving section 22 b receives theresponse information 4 a from the cleaning robot 300 a or the domesticapparatus. Thus, the receiving section 22 b outputs, to thecommunication possibility judging section 14, the response information 4a indicating that the cleaning robot 300 a or the domestic apparatus isin the state (1). On the other hand, in a case where the cleaning robot300 a or the domestic apparatus is in the state (2), the receivingsection 22 b does not receive the response information 4 a from thecleaning robot 300 a or the domestic apparatus. Namely, according to thefact that the response information 4 a is not supplied from the cleaningrobot 300 a or the domestic apparatus even after a predetermined periodof time has elapsed, the communication possibility judging section 14can determine that the cleaning robot 300 a or the domestic apparatus isin the state (2).

Herein, the cleaning robot 300 a is in the state (1). Therefore, uponreception of the response information 4 a from the cleaning robot 300 a,the receiving section 22 b outputs the response information 4 a to thecommunication possibility judging section 14. On the other hand, sincethe television image receiver 300 b is assumed to be in the state (2) asdescribed above, the receiving section 22 b does not receive theresponse information 4 a from the television image receiver 300 b.

In a case where the communication possibility judging section (secondjudging means) 14 receives the specifying information 1 a from theinformation extraction section 16 and the response information 4 a fromthe receiving section 22 b, the communication possibility judgingsection 14 outputs, to the information extraction section 16, a judgmentresult 6 b indicating that the cleaning robot 300 a or the domesticapparatus specified by the specifying information 1 a is communicablyconnected to the server 200 (i.e., the cleaning robot 300 a or thedomestic apparatus specified by the specifying information 1 a is in thestate (1)).

On the other hand, in a case where the communication possibility judgingsection 14 receives the specifying information 1 a from the informationextraction section 16 and does not receive the response information 4 afrom the receiving section 22 b, the communication possibility judgingsection 14 outputs, to the information extraction section 16, a judgmentresult 6 b indicating that the cleaning robot 300 a or the domesticapparatus specified by the specifying information 1 a is notcommunicably connected to the server 200 (i.e., the cleaning robot 300 aor the domestic apparatus specified by the specifying information 1 a isin the state (2)).

Upon reception of the image information 2 from the receiving section 22b, the information extraction section 16 extracts, from the image 5indicated by the image information 2, the specifying information 1 a andthe instruction information 1 b each of which is associated with theimage 5. Then, the information extraction section 16 outputs thespecifying information 1 a to the communication possibility judgingsection 14 and to the transmitting section 21 b, and outputs theinstruction information 1 b to the transmitting section 21 b.

Here, each of the specifying information 1 a and the instructioninformation 1 b is associated with the image 5 in the following manner.For example, the specifying information 1 a and the instructioninformation 1 b are stored in meta content (e.g., an Exif header) ofdata of the image 5.

Alternatively, the image 5 may be associated with the specifyinginformation 1 a and the instruction information 1 b by use of a name ofa path for storing the image 5. For example, a path“/image/cleaner/photo.jpg” may indicate an instruction to cause thecleaning robot 300 a to take a photograph, a path“/image/cleaner/moveForward.jpg” may indicate an instruction to move thecleaning robot 300 a forward, a path “/image/tv/on.jpg” may indicate aninstruction to turn on the television image receiver 300 b, and a path“/image/tv/off.jpg” may indicate an instruction to turn off thetelevision image receiver 300 b.

In a case where the judgment result 6 b supplied from the communicationpossibility judging section 14 indicates that the cleaning robot 300 aor the domestic apparatus specified by the specifying information 1 a isin the state (2), the information extraction section 16 may change thespecifying information 1 a and the instruction information 1 b.

For example, in a case where the specifying information 1 a specifiesthe “television image receiver 300 b” and the instruction information 1b instructs to “turn on”, the information extraction section 16 changesthe specifying information 1 a so that the specifying information 1 aspecifies the “cleaning robot 300 a” and changes the instructioninformation 1 b so that the instruction information 1 b gives aninstruction to “turn on the television image receiver 300 b”, since thetelevision image receiver 300 b is determined to be in the state (2).

Namely, the server 200 further includes the communication possibilityjudging section (second judging means) 14 for judging whether or not thedomestic apparatus (controlled apparatus) is communicably connected tothe server 200. In a case where the communication possibility judgingsection (second judging means) 14 determines that the domestic apparatus(controlled apparatus) is not connected to the server 200, thetransmitting section (first transmitting means) 21 b causes the cleaningrobot (another controlled apparatus) 300 a communicably connected to theserver 200 and to the domestic apparatus (controlled apparatus) to relaythe instruction information 1 b instructing execution of a predeterminedprocess to the domestic apparatus (controlled apparatus), so that theinstruction information 1 b instructing execution of the predeterminedprocess is transmitted to the domestic apparatus (controlled apparatus).

FIG. 2 does not show transmission/reception of any message (e.g., amessage contributed by user's entering of texts or a photographcontributed by a user) except for the image information 2 supplied tothe server 200. However, in a case where either of the specifyinginformation 1 a and the instruction information 1 b cannot be extractedby the information extraction section 16, the transmitting section 21 btransmits simply a message received by the receiving section 22 b sothat the message can be displayed by the smartphone 100.

[Configuration of Cleaning Robot 300 a]

With reference to (a) of FIG. 3, the following describes a configurationof the cleaning robot 300 a. (a) of FIG. 3 is a block diagramillustrating a main part of the configuration of the cleaning robot 300a. In the following description, functions of a communication section 20c (a transmitting section 21 c, a receiving section 22 c, and areceiving section 22 d) and a process executing section 15 a will beexplained in this order.

The communication section 20 c has the same function as that of thecommunication section 20 a, and therefore detailed description of thecommunication section 20 c is omitted here. The communication section 20c includes the transmitting section 21 c and the receiving section 22 c.

Upon reception of instruction information 1 b from the receiving section22 c, the transmitting section (second transmitting means) 21 ctransmits the instruction information 1 b to the domestic apparatus(television image receiver 300 b). Upon reception of a response request4 b from the receiving section 22 c, the transmitting section 21 ctransmits the response information 4 a or apparatus information 3 b tothe server 200. Further, upon reception of an execution result 3 a fromthe process executing section 15 a, the transmitting section 21 ctransmits the execution result 3 a to the server 200.

The receiving section (second receiving means, third receiving means) 22c receives, from the server 200, the instruction information 1 binstructing execution of a predetermined process. The receiving section22 c transmits, to the process executing section 15 a, the instructioninformation 1 b thus received. In a case where the instructioninformation 1 b is to be relayed to the domestic apparatus, thereceiving section 22 c outputs the instruction information 1 b also tothe transmitting section 21 c.

Namely, the cleaning robot (controlled apparatus) 300 a may furtherinclude (i) the receiving section (third receiving means) 22 c forreceiving information that the instruction information 1 b instructingexecution of the predetermined process is to be relayed and (ii) thetransmitting section (second transmitting means) 21 c for transmitting,in a case where the receiving section (third receiving means) 22 creceives the information that the relaying is to be carried out, theinstruction information 1 b that is received by the receiving section(second receiving means) 22 c and that instructs execution of thepredetermined process to the external television image receiver(controlled apparatus) 300 b indicated by this information.

According to the instruction information 1 b received by the receivingsection 22 c, the process executing section (executing means) 15 aexecutes the predetermined process (for example, a process of moving thecleaning robot 300 a forward by 1 m or a process of rotating thecleaning robot 300 a by 90° to the right; see (a) of FIG. 7). Theprocess executing section 15 a outputs, to the transmitting section 21c, a result of execution of the predetermined process (execution result3 a).

[Configuration of Television Image Receiver 300 b]

With reference to (b) of FIG. 3, the following describes a configurationof the television image receiver 300 b. (b) of FIG. 3 is a block diagramillustrating a main part of the configuration of the television imagereceiver 300 b.

A receiving section (second receiving means) 22 d receives, from thecleaning robot 300 a, instruction information 1 b instructing executionof a predetermined process, and outputs the instruction information 1 bto a process executing section 15 b.

According to the instruction information 1 b received by the receivingsection 22 d, the process executing section (executing means) 15 bexecutes the predetermined process (e.g., a process of turning on oroff; see (b) of FIG. 7).

[Example where a Plurality of Images 5 are Contributed at Once]

With reference to FIG. 8, the following describes an example where thecleaning robot 300 a or the domestic apparatus is caused to execute aseries of processes in response to user's contributing a plurality ofimages 5 at once. FIG. 8 is a view schematically illustrating the familybulletin board on which the cleaning robot 300 a contributes a result ofexecution of a series of processes specified by a plurality of images 5contributed by a user at once.

As shown in FIG. 8, the user can contribute at once three images 5respectively instructing the cleaning robot 300 a to “move forward by 1meter”, to “rotate by 90° to the right”, and to “take a photograph ofthe surroundings” on the family message board as a message 8 g.

Namely, in response to user's touch on the image selecting button 44(see FIG. 6), the smartphone 100 can display a plurality of imagesselectable by the user and transmit, to the server 200, a plurality ofpieces of image information 2 respectively indicating a plurality ofimages 5 selected from the plurality of images. Thus, the smartphone 100makes it possible to omit a user's troublesome procedure of contributingthe images 5 one after another repeatedly in order to give aninstruction to execute a series of processes.

Upon reception of the plurality of pieces of image information 2, theserver 200 transmits, to an apparatus(es) specified by pieces ofspecifying information 1 a respectively associated with the images 5indicated by their corresponding plurality of pieces of imageinformation 2, pieces of instruction information 1 b each giving aninstruction to execute a predetermined process.

In the example shown in FIG. 8, the server 200 transmits, to the“cleaning robot 300 a” specified by the pieces of specifying information1 a, three pieces of instruction information 1 b respectivelyinstructing processes of “moving forward by 1 m”, “rotating by 90° tothe right”, and “taking a photograph of the surroundings”.

Upon reception of the three pieces of instruction information 1 b, thecleaning robot 300 a executes these three processes in order. It shouldbe noted that the cleaning robot 300 a contributes, on the familymessage board, only the result of execution of the process of “taking aphotograph of the surroundings” as a message 8 h, without transmittingthe results of the execution of the processes of “moving forward by 1 m”and “rotating by 90° to the right” to the server 200.

Namely, in the case where the cleaning robot 300 a receives a pluralityof pieces of instruction information 1 b as a result of user'scontributing a plurality of images 5 at once on the family messageboard, the cleaning robot 300 a contributes only a final result thereonand does not contribute an interim result thereon. Thus, the cleaningrobot 300 a makes it possible to omit a user's troublesome procedure ofchecking the interim result.

If an apparatus receives pieces of instruction information 1 bcontradicting each other, the apparatus does not need to executepredetermined processes specified by such the pieces of instructioninformation 1 b. For example, in a case where the cleaning robot 300 areceives two pieces of instruction information 1 b instructing processesof “moving forward by 1 m” and “moving backward by 1 m”, the cleaningrobot 300 a may be in a standby mode without executing the processesspecified by the two pieces of instruction information 1 b. For anotherexample, in a case where the television image receiver 300 b receivestwo pieces of instruction information 1 b instructing processes of“turning on” and “turning off”, the television image receiver 300 b maybe kept off without executing the processes specified by the two piecesof instruction information 1 b.

In the example shown in FIG. 8, the user instructs only the cleaningrobot 300 a (i.e., the same apparatus) to execute the processes.Alternatively, instructions to execute the series of processes may begiven to different apparatuses.

Further alternatively, one image 5 may be associated with a plurality ofpieces of specifying information 1 a and a plurality of pieces ofinstruction information 1 b. In this case, in response to user'sselection of such the one image 5, the cleaning robot 300 a or thedomestic apparatus may be caused to execute a series of processes. Thus,the smartphone 100 makes it possible to avoid causing again and againthe user to carry out a troublesome procedure of selecting the image 5in order to give an instruction to execute the series of processes.

[Example where Image is Contributed Via Television Image Receiver 300 b]

In a case where the television image receiver 300 b is included in thenetwork, the user can contribute an image 5 via the television imagereceiver 300 b.

Namely, in response to user's selection of an image 5 via an interface(e.g., a remote controller) capable of operating the television imagereceiver 300 b, the television image receiver 300 b (control apparatus)transmits, to the server 200, image information 2 indicative of theimage 5, so as to cause an apparatus (e.g., the cleaning robot 300 a orany of the domestic apparatuses such as the air conditioner 300 c)specified by specifying information 1 a associated with the image 5 toexecute a predetermined process specified by instruction information 1 bassociated with the image 5.

Thus, the television image receiver 300 b makes it possible to providemore convenience use, for example, by which a user can turn on the airconditioner 300 c located on the first floor while the user is watchinga television on the second floor.

[Process Executed in Family Message Board System 400]

With reference to FIG. 9, the following describes a flow of execution ofa process in the family message board system 400. FIG. 9 is a flow chartof an example of a process executed in the family message board system400 (the smartphone 100, the server 200, and the cleaning robot 300 a).In the following description, the expression “a step of . . . ” inparentheses indicates one of steps included in a method of controllingthe smartphone 100.

The image display section 71 a displays images 5 in such a manner as tobe selectable by a user, each of the images 5 being associated with (i)specifying information 1 a (herein, indicating the “cleaning robot 300a”) and (ii) instruction information 1 b (herein, indicating aninstruction to “take a photograph of the surroundings”) (step 1:hereinafter abbreviated as “S1”, and corresponding to a step ofdisplaying). In a case where the input section 40 (the input screen 41,the input control section 42) determines that one of the images 5 isselected by the user (YES in S2), the process instruction section 11transmits, to the server 200, image information 2 indicative of theimage 5 thus selected (S3, a step of instruction).

In a case where the receiving section 22 b in the server 200 receivesthe image information 2 (S4), the transmitting section 21 b transmits,to a plurality of smartphones 100, contribution information 4 c of amessage including the image information 2 (S24). Receiving sections 22 aof the respective smartphones 100 receive the contribution information 4c (S25), and list display sections 71 b of the respective smartphones100 each display the message in such a manner as to be viewed in a list(S26).

The information extraction section 16 in the server 200 attempts toextract the specifying information 1 a and the instruction information 1b associated with the image 5 indicated by the image information 2 (S5).In a case where the specifying information 1 a and the instructioninformation 1 b are extracted from the image 5 by the informationextraction section 16 (YES in S27), the transmitting section 21 btransmits a response request 4 b to an apparatus (herein, the cleaningrobot 300 a) indicated by the specifying information 1 a thus extracted(S6).

In a case where at least one of the specifying information 1 a and theinstruction information 1 b is not extracted from the image 5 by theinformation extraction section 16 (NO in S27), the server 200 regardsthe image 5 as a general contributed message. Then, steps following S27are skipped and the procedure ends.

In a case where the receiving section 22 c in the cleaning robot 300 areceives the response request 4 b (S7), the transmitting section 21 ctransmits response information 4 a and apparatus information 3 b to theserver 200 (S8).

In a case where the receiving section 22 b in the server 200 receivesthe response information 4 a and the apparatus information 3 b (S9), thecommunication possibility judging section 14 determines that thecleaning robot 300 a is in a communicable state (in the state (1)) (YESin S10), and the transmitting section 21 b transmits the instructioninformation 1 b to the cleaning robot 300 a (S11). Further, thetransmitting section 21 b transmits the apparatus information 3 b to thesmartphones 100 (S12). The receiving sections 22 a in the respectivesmartphones 100 receive the apparatus information 3 b (S13).

In a case where the receiving section 22 c in the cleaning robot 300 areceives the instruction information 1 b (S14), the process executingsection 15 a executes a predetermined process specified by theinstruction information 1 b (S15). Then, the transmitting section 21 ctransmits an execution result 3 a to the server 200 (S16).

In a case where the receiving section 22 b in the server 200 receivesthe execution result 3 a (S17), the transmitting section 21 b transmitsthe execution result 3 a to the smartphones 100 (S18). The receivingsections 22 a in the respective smartphones 100 receive the executionresult 3 a (S19). The smartphones 100 each display, in the vicinity ofthe message, a taken image obtained from the cleaning robot 300 a viathe server 200 (see (b) of FIG. 6).

With reference to FIG. 10, the following describes another flow ofexecution of a process in the family message board system 400. FIG. 10is a flow chart of another example of a process executed in the familymessage board system 400 (the smartphone 100, the server 200, thecleaning robot 300 a, and the television image receiver 300 b). Notethat some of steps in FIG. 10 which are the same as those described withreference to FIG. 9 are given the same reference numerals, anddescriptions thereof are omitted (descriptions of S1 to S5, S11 to S14,S16 to S19, and S24 to S27 are omitted).

The transmitting section 21 b in the server 200 transmits a responserequest 4 b to an apparatus (herein, the television image receiver 300b) indicated by the specifying information 1 a which is extracted (S6).However, as described above, the television image receiver 300 b is notconnected to the network. Therefore, the response request 4 b is notreceived by the television image receiver 300 b, and thus the responseinformation 4 a cannot be obtained from the television image receiver300 b.

Based on the fact that the response information 4 a is not supplied fromthe television image receiver 300 b even after a predetermined period oftime has elapsed, the communication possibility judging section 14determines that the television image receiver 300 b is in the state (2)(NO in S10).

For example, in a case where the specifying information 1 a specifiesthe “television image receiver 300 b” and the instruction information 1b gives an instruction to “turn on”, the information extraction section16 changes the specifying information 1 a and the instructioninformation 1 b so that the specifying information 1 a specifies the“cleaning robot 300 a” and the instruction information 1 b gives aninstruction to “turn on the television image receiver 300 b” (S20).

The transmitting section 21 c in the cleaning robot 300 a transmits theinstruction information 1 b thus changed to the television imagereceiver 300 b (S21). In a case where the receiving section 22 d in thetelevision image receiver 300 b receives the instruction information 1 b(S22), the process executing section 15 b executes a predeterminedprocess specified by the instruction information 1 b (S23).

In FIGS. 9 and 10, the server 200 is to receive the image information 2in S4, and subsequently the server 200 is to transmit the responserequest 4 b in S6. In FIG. 9, the server 200 is then to receive theresponse information 4 a and the apparatus information 3 b in S9.However, timings of the reception and the transmission are not limitedto these. For example, regardless of whether or not the imageinformation 2 is received, the server 200 may transmit the responserequest 4 b at regular intervals, or the cleaning robot 300 a maytransmit the apparatus information 3 b or information being similar tothe response information 4 a at regular intervals.

[Effects Given by Smartphone 100]

The smartphone 100 enables a user to control the cleaning robot 300 a orthe domestic apparatus, which is a control target, even if the user doesnot carry out an explicit operation of specifying the cleaning robot 300a or the domestic apparatus. With this, the smartphone 100 providesimproved user's convenience.

SUMMARY

A control apparatus (the smartphone 100 or the television image receiver300 b) according to one embodiment of the present invention is (1) acontrol apparatus for controlling a controlled apparatus via a server,including: (2) first display means for displaying images in such amanner as to be selectable by a user, each of the images beingassociated with (i) a piece of specifying information for specifying acontrolled apparatus and (ii) a piece of instruction information forinstructing the controlled apparatus to execute a predetermined process;and (3) instruction means for transmitting, in a case where the userselects at least one of the images displayed by the first display means,a piece of image information indicative of said at least one of theimages to the server, so as to cause a controlled apparatus that isspecified by a piece of specifying information associated with said atleast one of the images to execute a predetermined process that isinstructed by a piece of instruction information associated with said atleast one of the images.

A method for controlling a control apparatus according to one embodimentof the present invention is (1) a method for controlling a controlapparatus that controls a controlled apparatus via a server, including:(2) a first displaying step of displaying images in such a manner as tobe selectable by a user, each of the images being associated with (i) apiece of specifying information for specifying a controlled apparatusand (ii) a piece of instruction information for instructing thecontrolled apparatus to execute a predetermined process; and (3) aninstruction step of transmitting, in a case where the user selects atleast one of the images displayed in the first displaying step, a pieceof image information indicative of said at least one of the images tothe server, so as to cause a controlled apparatus that is specified by apiece of specifying information associated with said at least one of theimages to execute a predetermined process that is instructed by a pieceof instruction information associated with said at least one of theimages.

According to the above-described conventional technique, informationrelated to an external apparatus which is to be controlled (e.g., atelevision image receiver) is not included in image data, and thereforea user needs to specify the external apparatus. Namely, the user isforced to carry out an explicit operation of specifying an externalapparatus which is to be controlled and transmitting an instruction tothe external apparatus thus specified.

However, what is truly required by the user is a “function” that isprovided by the external apparatus. In some cases, the apparatus thatplays a main role in order to provide the “function” may be arbitraryone. Thus, the interface always forcing the user to carry out such theexplicit operation is not user-friendly.

On the other hand, according to the above features (1) through (3), thecontrol apparatus of the one embodiment of the present invention and themethod for controlling that control apparatus transmits, to the server,the image information indicative of the image associated with (i) thespecifying information for specifying the controlled apparatus and (ii)the instruction information for instructing the controlled apparatus toexecute the predetermined process. Namely, the control apparatus or thelike always transmits the image information to the server, and thereforedoes not require the user to carry out the explicit operation for“specifying the external apparatus”.

In other words, the control apparatus or the like allows the user tocontrol the controlled apparatus even if the user does not carry out theexplicit operation of specifying the controlled apparatus, which is thecontrol target. With this, the control apparatus or the like providesimproved user's convenience.

Further, the control apparatus of the one embodiment of the presentinvention may be configured such that: (1) the instruction meanstransmits the piece of image information to the server providing aservice which enables information to be shared by users having beenregistered in advance; and (2) said control apparatus further includessecond display means for displaying, as information having been sharedby the users, pieces of information in time series in such a manner asto be viewed at a glance, the pieces of information including at leastthe piece of image information transmitted by the instruction means.

According to the above features (1) and (2), the control apparatus ofthe one embodiment of the present invention transmits the imageinformation to the server providing the service (e.g., a social networkservice) that enables information to be shared by users. Therefore, thecontrol apparatus allows the image information to be shared by theplurality of users who uses the service.

Further, the control apparatus displays a message(s) contributed by auser(s), the image information transmitted in response to aninstruction(s) given by a user(s), and/or the like in time series insuch a manner as to be viewed at a glance. This allows all the users toknow “who” has transmitted “what kind of instruction” to “whom”. Withthis, the control apparatus provides further improved user'sconvenience.

Further, the control apparatus of the one embodiment of the presentinvention may further include: (1) obtaining means for obtaining, fromthe server, a result of execution of the predetermined process carriedout by the controlled apparatus, (2) in a case where the obtaining meansobtains the result of execution, the second display means furtherdisplaying the result of execution in the vicinity of the pieces ofinformation displayed in such the manner as to be viewed at a glance.

According to the above features (1) and (2), the control apparatus ofthe one embodiment of the present invention allows the user to know thatthe result of execution is a response to the pieces of informationdisplayed in such the manner as to be viewed at a glance. With this, thecontrol apparatus provides further improved user's convenience.

Further, the control apparatus of the one embodiment of the presentinvention may be configured such that (1) in a case where a user selectsa plurality of ones of the images, the instruction means transmits aplurality of pieces of image information indicative of the respectiveplurality of ones of the images to the server, so as to cause acontrolled apparatus specified by a plurality of pieces of specifyinginformation associated with the respective plurality of ones of theimages to sequentially execute predetermined processes instructed by aplurality of pieces of instruction information associated with therespective plurality of ones of the images.

According to the above feature (1), the control apparatus of the oneembodiment of the present invention instructs the controlled apparatusto execute a series of processes, thereby omitting a user's troublesomeprocedure of selecting images one after another repeatedly. With this,the control apparatus provides further improved user's convenience.

Further, the control apparatus of the one embodiment of the presentinvention may further include: (1) first judging means for judgingwhether or not the controlled apparatus is able to execute thepredetermined process(es), (2) in a case where the first judging meansdetermines that the controlled apparatus is not able to execute thepredetermined process(es), the first display means not displaying theimage(s) associated with the predetermined process(es).

According to the above feature (1), the control apparatus of the oneembodiment of the present invention can limit a range of the imagesselectable by the user, according to the judgment on whether or not thecontrolled apparatus is capable of executing the predeterminedprocess(es). Thus, the control apparatus can reduce user's burden ofselection, thereby providing further improved user's convenience.

A server (server 200) of one embodiment of the present invention may be(1) a server communicably connected to the above control apparatus,including: (2) first receiving means for receiving, from the controlapparatus, a piece of image information indicative of an image selectedby a user; and (3) first transmitting means for transmitting, to acontrolled apparatus specified by a piece of specifying informationassociated with the image indicated by the piece of image informationreceived by the first receiving means, a piece of instructioninformation for instructing the controlled apparatus to execute apredetermined process.

According to the above features (1) through (3), the server of the oneembodiment of the present invention relays the instruction informationto the controlled apparatus specified by the specifying information.With this, the server can avoid control apparatus's requiring the userto carry out the above-described explicit operation. With this, theserver provides improved user's convenience.

A controlled apparatus (the cleaning robot 300 a, the television imagereceiver 300 b, the air conditioner 300 c, the illumination apparatus300 d, or the video recording apparatus 300 e) of one embodiment of thepresent invention may be (1) a controlled apparatus controlled by theabove control apparatus via the above server, including: (2) secondreceiving means for receiving a piece of instruction information forinstructing the controlled apparatus to execute a predetermined process;and (3) executing means for executing the predetermined processaccording to the piece of instruction information received by the secondreceiving means.

According to the above features (1) through (3), the controlledapparatus of the one embodiment of the present invention can execute thepredetermined process instructed by the piece of instruction informationthat is relayed by the server. With this, the controlled apparatus canexecute the predetermined process instructed by the control apparatus,without causing the control apparatus to require the user to carry outthe above-described explicit operation. With this, the controlledapparatus provides improved user's convenience.

A control system (family message board system 400) of one embodiment ofthe present invention may include (1) the above control apparatus, (2)the above server, and (3) the above controlled apparatus.

According to the above features (1) through (3), the control system ofthe one embodiment of the present invention provides the same effects asthose attained by the control apparatus, the server, or the controlledapparatus.

Note that each of the control apparatus, the server, and the controlledapparatus may be realized by a computer. In this case, the presentinvention encompasses (i) a control program for causing the computer tooperate as each means of the control apparatus, the server, and thecontrolled apparatus so as to realize the control apparatus, the server,or the controlled apparatus by the computer and (ii) a computer-readablerecording medium in which the control program is stored.

[Combination of Features (Technical Means) in Embodiments]

It should be noted that the features included in the above-describedembodiments can be combined as appropriate. Namely, the whole of or apart of all the features explained in the above-described embodimentsmay be combined not only in the explained embodiments but also in otherembodiments, and embodiments thus obtained are also encompassed in thetechnical scope of the present invention. Further, it is possible tocombine any of technical features described in the embodiments of thepresent invention in order to provide a new technical scheme.

[Examples of Realization by Software]

Finally, the blocks of the smartphone 100, the server 200, the cleaningrobot 300 a, and the television image receiver 300 b may be realized byway of hardware as executed by a logical circuit provided on anintegrated circuit (IC chip) or software as executed by a CPU (CentralProcessing Unit).

In the latter case, each of the smartphone 100, the server 200, thecleaning robot 300 a, and the television image receiver 300 b includes,e.g., a CPU for executing instructions in programs realizing thefunctions, a ROM (Read Only Memory) which contains the programs, a RAM(Random Access Memory) to which the programs are loaded, and a recordingmedium such as a memory containing the programs and various data. Theobject of the present invention can also be achieved by mounting, to thesmartphone 100, the server 200, the cleaning robot 300 a, or thetelevision image receiver 300 b, a computer-readable storage mediumcontaining control program code (executable program, intermediate codeprogram, or source program) for the smartphone 100, the server 200, thecleaning robot 300 a, or the television image receiver 300 b which issoftware realizing the aforementioned functions, in order for thecomputer (or CPU, MPU) to retrieve and execute the program codecontained in the storage medium.

The storage medium may be a non-transitory tangible medium, for example,a tape such as a magnetic tape or a cassette tape; a disk such as amagnetic disk (e.g., a floppy (Registered Trademark) disk or a harddisk) or an optical disk (e.g., CD-ROM, MO, MD, DVD, or CD-R); a cardsuch as an IC card (memory card) or an optical card; a semiconductormemory such as a mask ROM, EPROM, EEPROM (Registered Trademark), or aflash ROM; or a logical circuit such as PLD (Programmable Logic Device)or FPGA (Field Programmable Gate Array).

The smartphone 100, the server 200, the cleaning robot 300 a, or thetelevision image receiver 300 b may be arranged to be connectable to acommunications network so that the program code may be delivered overthe communications network. The communications network only needs to bethe one via which the program code can be delivered, and is not limitedin any particular manner. The communications network may be, forexample, the Internet, an intranet, extranet, LAN, ISDN, VAN, CATVcommunications network, virtual dedicated network (virtual privatenetwork), telephone line network, mobile communications network, orsatellite communications network. The transfer medium which makes up thecommunications network only needs to be the one by which the programcode can be delivered, and is not limited to any particular arrangementor type. The transfer medium may be, for example, wired line such asIEEE 1394, USB, electric power line, cable TV line, telephone line, orADSL line; or wireless such as infrared radiation (IrDA, remotecontrol), Bluetooth (Registered Trademark), IEEE 802.11 wireless, HDR(High Data Rate), NFC (Near Field Communication), DLNA (Digital LivingNetwork Alliance), mobile telephone network, satellite line, orterrestrial digital network. The present invention encompasses computerdata signals embedded in a carrier wave in which the program code isembodied electronically.

As such, “means” herein does not necessarily refer to physical means,but also refers to a case where a function of each means is realized bysoftware. Further, a function of one means may be realized by two ormore physical means, or functions of two or more means may be realizedby one physical means.

INDUSTRIAL APPLICABILITY

The present invention is applicable to, e.g., apparatuses (such as asmartphone, a tablet terminal, a personal computer, and a televisionimage receiver) capable of controlling a controlled apparatus via aserver.

REFERENCE SIGNS LIST

-   -   1 a Specifying information (specifying information)    -   1 b Instruction information (instruction information)    -   2 Image information (image information)    -   3 a Execution result (result of execution)    -   5 Image (image)    -   11 Process instruction section (instruction means)    -   12 Execution possibility judging section (first judging means)    -   13 Execution result obtaining section (obtaining means)    -   14 Communication possibility judging section (second judging        means)    -   15 a Process executing section (executing means)    -   15 b Process executing section (executing means)    -   21 b Transmitting section (first transmitting means)    -   21 c Transmitting section (second transmitting means)    -   22 b Receiving section (first receiving means)    -   22 c Receiving section (second receiving means, third receiving        means)    -   22 d Receiving section (second receiving means)    -   71 a Image display section (first display means)    -   71 b List display section (second display means)    -   100 Smartphone (control apparatus)    -   200 Server (server)    -   300 a Cleaning robot (controlled apparatus)    -   300 b Television image receiver (control apparatus, controlled        apparatus)    -   300 c Air conditioning apparatus (controlled apparatus)    -   300 d Illumination apparatus (controlled apparatus)    -   300 e Video recording apparatus (controlled apparatus)    -   400 Family message board system (control system)

The invention claimed is:
 1. A control apparatus for controlling acontrolled apparatus via a server, comprising: first display means fordisplaying images in such a manner as to be selectable by a user, eachof the images being associated with (i) a piece of specifyinginformation for specifying a controlled apparatus and (ii) a piece ofinstruction information for instructing the controlled apparatus toexecute a predetermined process; and instruction means for transmitting,in a case where the user selects at least one of the images displayed bythe first display means, a piece of image information indicative of saidat least one of the images to the server, so as to cause a controlledapparatus that is specified by a piece of specifying informationassociated with said at least one of the images to execute apredetermined process that is instructed by a piece of instructioninformation associated with said at least one of the images, wherein:the instruction means transmits the piece of image information to theserver providing a social network service which enables information tobe shared by users having been registered in advance; and said controlapparatus further comprises: second display means for displaying, asinformation having been shared by all the users who have been registeredin advance, pieces of information including at least the piece of imageinformation transmitted by the instructions means; and obtaining meansfor obtaining, from the server, a result of execution of thepredetermined process carried out by the controlled apparatus, in a casewhere the obtaining means obtains the result of execution, the seconddisplay means further displaying, as information having been shared byall the user who have been registered in advance, the result ofexecution in the vicinity of the pieces of information displayed in sucha manner as to be viewed at a glance, in a case where a user selects aplurality of ones of the images, the instruction means transmits aplurality of pieces of image information indicative of the respectiveplurality of ones of the images to the server, so as to cause acontrolled apparatus specified by a plurality of pieces of specifyinginformation associated with the respective plurality of ones of theimages to sequentially execute predetermined processes instructed by aplurality of pieces of instruction information associated with therespective plurality of ones of the images.
 2. The control apparatus asset forth in claim 1, further comprising: first judging means forjudging whether or not the controlled apparatus is able to execute thepredetermined process, in a case where the first judging meansdetermines that the controlled apparatus is not able to execute thepredetermined process, the first display means not displaying an imagewhich is associated with the predetermined process and is included inthe images.
 3. A server communicably connected to a control apparatus asset forth in claim 1, comprising: first receiving means for receiving,from the control apparatus, a piece of image information indicative ofan image selected by a user; and first transmitting means fortransmitting, to a controlled apparatus specified by a piece ofspecifying information associated with the image indicated by the pieceof image information received by the first receiving means, a piece ofinstruction information for instructing the controlled apparatus toexecute a predetermined process.
 4. A controlled apparatus controlled bya control apparatus as set forth in claim 1 via a server communicablyconnected to the control apparatus, the server including (i) firstreceiving means for receiving, from the control apparatus, a piece ofimage information indicative of an image selected by a user and (ii)first transmitting means for transmitting, to a controlled apparatusspecified by a piece of specifying information associated with the imageindicated by the piece of image information received by the firstreceiving means, a piece of instruction information for instructing thecontrolled apparatus to execute a predetermined process, said controlledapparatus comprising: second receiving means for receiving the piece ofinstruction information for instructing the controlled apparatus toexecute the predetermined process; and executing means for executing thepredetermined process according to the piece of instruction informationreceived by the second receiving means.
 5. A control system, comprising:a control apparatus as set forth in claim 1; a server communicablyconnected to the control apparatus, the server including (i) firstreceiving means for receiving, from the control apparatus, a piece ofimage information indicative of an image selected by a user and (ii)first transmitting means for transmitting, to a controlled apparatusspecified by a piece of specifying information associated with the imageindicated by the piece of image information received by the firstreceiving means, a piece of instruction information for instructing thecontrolled apparatus to execute a predetermined process; and acontrolled apparatus controlled by the control apparatus via the server,the controlled apparatus including (i) second receiving means forreceiving the piece of instruction information for instructing thecontrolled apparatus to execute the predetermined process; and (ii)executing means for executing the predetermined process according to thepiece of instruction information received by the second receiving means.6. A method for controlling a control apparatus that controls acontrolled apparatus via a server, comprising: a first displaying stepof displaying images in such a manner as to be selectable by a user,each of the images being associated with (i) a piece of specifyinginformation for specifying a controlled apparatus and (ii) a piece ofinstruction information for instructing the controlled apparatus toexecute a predetermined process; and an instruction step oftransmitting, in a case where the user selects at least one of theimages displayed in the first displaying step, a piece of imageinformation indicative of said at least one of the images to the server,so as to cause a controlled apparatus that is specified by a piece ofspecifying information associated with said at least one of the imagesto execute a predetermined process that is instructed by a piece ofinstruction information associated with said at least one of the images,wherein in the instruction step, the piece of image information istransmitted to the server providing a social network service whichenables information to be shared by users having been registered inadvance; and said method further comprises: a second displaying step ofdisplaying, as information having been shared by all the users who havebeen registered in advance, pieces of information including at least thepiece of image information transmitted by the instruction step; and anobtaining step of obtaining, from the server, a result of execution ofthe predetermined process carried out by the controlled apparatus, in acase where the result of execution is obtained in the obtained step, theresult of execution is further displayed in the second displaying step,as information having been shared by all the users who have beenregistered in advance, in the vicinity of the pieces of informationdisplayed in such a manner as to be viewed at a glance, in a case wherea user selects a plurality of ones of the images, the instruction steptransmits a plurality of pieces of image information indicative of therespective plurality of ones of the images to the server, so as to causea controlled apparatus specified by a plurality of pieces of specifyinginformation associated with the respective plurality of ones of theimages to sequentially execute predetermined processes instructed by aplurality of pieces of instruction information associated with therespective plurality of ones of the images.
 7. A non-transitorycomputer-readable storage medium storing therein a control program forcausing a computer to function as a control apparatus for controlling acontrolled apparatus via a server, said control program causing thecomputer to execute: a first displaying step of displaying images insuch a manner as to be selectable by a user, each of the images beingassociated with (i) a piece of specifying information for specifying acontrolled apparatus and (ii) a piece of instruction information forinstructing the controlled apparatus to execute a predetermined process;and an instruction step of transmitting, in a case where the userselects at least one of the images displayed in the first displayingstep, a piece of image information indicative of said at least one ofthe images to the server, so as to cause a controlled apparatus that isspecified by a piece of specifying information associated with said atleast one of the images to execute a predetermined process that isinstructed by a piece of instruction information associated with said atleast one of the images, wherein in the instruction step, the piece ofimage information is transmitted to the server providing a socialnetwork service which enables information to be shared by users havingbeen registered in advance; and said control program further causing thecomputer to execute: a second displaying step of displaying, asinformation having been shared by all the users who have been registeredin advance, pieces of information including at least the piece of imageinformation transmitted by the instruction step; and an obtaining stepof obtaining, from the server, a result of execution of thepredetermined process carried out by the controlled apparatus, in a casewhere the result of execution is obtained in the obtained step, theresult of execution is further displayed in the second displaying step,as information having been shared by all the users who have beenregistered in advance, in the vicinity of the pieces of informationdisplayed in such a manner as to be viewed at a glance, in a case wherea user selects a plurality of ones of the images, the instruction steptransmits a plurality of pieces of image information indicative of therespective plurality of ones of the images to the server, so as to causea controlled apparatus specified by a plurality of pieces of specifyinginformation associated with the respective plurality of ones of theimages to sequentially execute predetermined processes instructed by aplurality of pieces of instruction information associated with therespective plurality of ones of the images.